Method for estimating examinee attribute parameters in cognitive diagnosis models

ABSTRACT

A method of determining a mastery level for an examinee from an assessment is disclosed. The method includes receiving one or more of an overall skill level for an examinee, a weight for the overall skill level, a covariate vector for an examinee, and a weight for the covariate vector. An examinee attribute value is computed using one or more of the received values for each examinee and each attribute. The computation of the examinee attribute values can include estimating the value using a Markov Chain Monte Carlo estimation technique. Examinee mastery levels are then assigned based on each examinee attribute level. Dichotomous or polytomous levels can be assigned based on requirements for the assessment.

TECHNICAL FIELD

The present invention relates generally to the field of assessmentevaluation. In particular, the invention relates to a method and systemfor evaluating assessment examinees on a plurality of attributes basedon responses to assessment items.

BACKGROUND

Standardized testing is prevalent in the United States today. Suchtesting is often used for higher education entrance examinations andachievement testing at the primary and secondary school levels. Theprevalence of standardized testing in the United States has been furtherbolstered by the No Child Left Behind Act of 2001, which emphasizesnationwide test-based assessment of student achievement.

The typical focus of research in the field of assessment measurement andevaluation has been on methods of item response theory (IRT). A goal ofIRT is to optimally order examinees along a low dimensional plane(typically unidimensional) based on the examinee's responses and thecharacteristics of the test items. The ordering of examinees is done viaa set of latent variables presupposed to measure ability. The itemresponses are generally considered to be conditionally independent ofeach other.

The typical IRT application uses a test to estimate an examinee's set ofabilities (such as verbal ability or mathematical ability) on acontinuous scale. An examinee receives a scaled score (a latent traitscaled to some easily understood metric) and/or a percentile rank. Thefinal score (an ordering of examinees along a latent dimension) is usedas the standardized measure of competency for an area-specific ability.

Although achieving a partial ordering of examinees remains an importantgoal in some settings of educational measurement, the practicality ofsuch methods is questionable in common testing applications. For eachexaminee, the process of acquiring the knowledge that each test purportsto measure seems unlikely to occur via this same low dimensionalapproach of broadly defined general abilities. This is, at least inpart, because such testing can only assess a student's abilitiesgenerally, but cannot adequately determine whether a student hasmastered a particular ability or not.

Because of this limitation, cognitive modeling methods, also known asskills assessment or skills profiling, have been developed for assessingstudents' abilities. Cognitive diagnosis statistically analyzes theprocess of evaluating each examinee on the basis of the level ofcompetence on an array of skills and using this evaluation to makerelatively fine-grained categorical teaching and learning decisionsabout each examinee. Traditional educational testing, such as the use ofan SAT score to determine overall ability, performs summativeassessment. In contrast, cognitive diagnosis performs formativeassessment, which partitions answers for an assessment examination intofine-grained (often discrete or dichotomous) cognitive skills orabilities in order to evaluate an examinee with respect to his level ofcompetence for each skill or ability. For example, if a designer of analgebra test is interested in evaluating a standard set of algebraattributes, such as factoring, laws of exponents, quadratic equationsand the like, cognitive diagnosis attempts to evaluate each examineewith respect to each such attribute. In contrast, summative analysissimply evaluates each examinee with respect to an overall score on thealgebra test.

Numerous cognitive diagnosis models have been developed to attempt toestimate examinee attributes. In cognitive diagnosis models, the atomiccomponents of ability, the specific skills that together comprise thelatent space of general ability, are referred to as attributes.Examinees are either masters of attributes or non-masters of attributes.The space of all attributes relevant to an examination is represented bythe set {(α₁, . . . , α_(k)}. Given a test with items j=1, . . . , J,the attributes necessary for each item can be represented in a matrix ofsize J×K. This matrix is referred to as a Q-matrix having valuesQ={q_(jk)}, where q_(jk)=1 when attribute k is required by item j andq_(jk)=0 when attribute k is not required by item j. Typically, theQ-matrix is constructed by experts and is pre-specified at the time ofthe examination analysis.

Cognitive diagnosis models can be sub-divided into two classifications:compensatory models and conjunctive models. Compensatory models allowfor examinees who are non-masters of one or more attributes tocompensate by being masters of other attributes. An exemplarycompensatory model is the common factor model. Low scores on somefactors can be compensated by high scores on other factors.

Numerous compensatory cognitive diagnosis models have been proposedincluding: (1) the Linear Logistic Test Model (LLTM) which modelscognitive facets of each item, but does not provide informationregarding the attribute mastery of each examinee; (2) the MulticomponentLatent Trait Model (MLTM) which determines the attribute features foreach examinee, but does not provide information regarding items; (3) theMultiple Strategy MLTM which can be used to estimate examineeperformance for items having multiple solution strategies; and (4) theGeneral Latent Trait Model (GLTM) which estimates characteristics of theattribute space with respect to examinees and item difficulty.

Conjunctive models, on the other hand, do not allow for compensationwhen critical attributes are not mastered. Such models more naturallyapply to cognitive diagnosis due to the cognitive structure defined inthe Q-matrix and will be considered herein. Such conjunctive cognitivediagnosis models include: (1) the DINA (deterministic inputs, noisy“AND” gate) model which requires the mastery of all attributes by theexaminee for a given examination item; (2) the NIDA (noisy inputs,deterministic “AND” gate) model which decreases the probability ofanswering an item for each attribute that is not mastered; (3) theDisjunctive Multiple Classification Latent Class Model (DMCLCM) whichmodels the application of non-mastered attributes to incorrectlyanswered items; (4) the Partially Ordered Subset Models (POSET) whichinclude a component relating the set of Q-matrix defined attributes tothe items by a response model and a component relating the Q-matrixdefined attributes to a partially ordered set of knowledge states; and(5) the Unified Model which combines the Q-matrix with terms intended tocapture the influence of incorrectly specified Q-matrix entries.

The Unified Model specifies the probability of correctly answering anitem X_(ij) for a given examinee i, item j, and set of attributes k=1, .. . , K as: $\begin{matrix}{{P\left( {{X_{ij} = {1❘\alpha_{i}}},\theta_{i}} \right)} = {\left( {1 - p} \right)\left\lbrack {{d_{j}{\prod\limits_{k = 1}^{K}{\pi_{jk}^{\alpha_{ik}{xq}_{jk}}r_{jk}^{({1 - {\alpha_{ik}{xq}_{jk}}})}{P_{j}\left( {\theta_{i} + {\Delta\quad c_{j}}} \right)}}}} +} \right.}} \\{\left. {\left( {1 - d_{j}} \right){P_{j}\left( \theta_{i} \right)}} \right\rbrack,{where}}\end{matrix}$θ_(i) is the latent trait of examinee i; p is the probability of anerroneous response by an examinee that is a master; d_(j) is theprobability of selecting the pre-defined Q-matrix strategy for item j;π_(jk) is the probability of correctly applying attribute k to item jgiven mastery of attribute k; r_(jk) is the probability of correctlyapplying attribute k to item j given non-mastery of attribute k; α_(ik)is an examinee attribute mastery level, and c_(j) is a value indicatingthe extent to which the Q-matrix entry for item j spans the latentattribute space.

One problem with the Unified Model is that the number of parameters peritem is unidentifiable. Accordingly, the Reparameterized Unified Model(RUM) was developed in an attempt to reparameterize the Unified Model ina manner consistent with the original interpretation of the modelparameters. For a given examinee i, item j, and Q-matrix defined set ofattributes k=1, . . . , K, the RUM specifies the probability ofcorrectly answering item X_(ij) as:${{P\left( {{X_{ij}❘\alpha_{i}},\theta_{i}} \right)} = {\pi_{j}^{*}{\prod\limits_{k = 1}^{K}{r_{jk}^{*{({1 - \alpha_{ik}})}{xq}_{jk}}{P_{c_{j}}\left( \theta_{i} \right)}}}}},{{{where}\quad\pi_{j}^{*}} = {\prod\limits_{k = 1}^{K}\pi_{jk}^{q_{jk}}}}$(the probability of correctly applying all K Q-matrix specifiedattributes for item j), $r_{jk}^{*} = \frac{r_{jk}}{\pi_{jk}}$(the penalty imposed for not mastering attribute k), and${P_{c_{j}}\left( \theta_{i} \right)} = \frac{{\mathbb{e}}^{({\theta_{i} + c_{j}})}}{1 + {\mathbb{e}}^{({\theta_{i} + c_{j}})}}$(a measure of the completeness of the model).

The RUM is a compromise of the Unified Model parameters that allow theestimation of both latent examinee attribute patterns and test itemparameters. The Fusion Model adds a Hierarchical Bayes Markov ChainMonte Carlo (MCMC) algorithm to estimate the parameters of the model.The item parameters in the Fusion model have a prior distribution thatis a Beta distribution, β(a, b), where (a, b) are defined for each setof item parameters, π*, r*, and c/3. Each set of hyperparameters is thenestimated within the MCMC chain to determine the shape of the priordistribution.

A difference between the Fusion Model and the RUM is that the α_(ik)term is replaced in the Fusion Model with a binary indicator function,I({overscore (α)}_(ik)>κ_(k)), where {overscore (α)}_(ik) is theunderlying continuous variable of examinee i for attribute k (i.e., anexaminee attribute value), and κ_(k) is the mastery threshold value that{overscore (α)}_(ik) must exceed for α_(ik)=1.

The set of K latent attributes can be considered to form a latent classmodel with 2^(K) classes. Alternatively, the set of attributes can beconsidered as defining 2^(K) different latent classes with membership ineach latent class estimated for each examinee. It should be noted thatcognitive diagnosis models are not entirely analogous to a typicallatent class model in that each Q-matrix defines a limited relationshipbetween the set of attributes and the set of test items. For each of theUnified Model, the RUM and Fusion Model, the item response function isnot only a function of class membership, but also a function of thecontinuous individual examinee ability parameter bounded by the itemresponse probability defined by the latent class.

Latent class models and cognitive diagnosis models differ in the meaningof class membership. A latent class pattern can be written as a sequenceof binary digits. For example, the latent class “1101” indicates afour-variable Q-matrix with the members of the class being masters ofattributes one, two and four and non-masters of attribute three. Latentclass models consider class membership by the entire label (i.e., alatent class model would estimate a probability of correct response forexaminees lacking the third attribute). In contrast, informationpertaining to each of the digits are of interest in the cognitivediagnosis model as educators seek to move each examinee from theircurrent class to the class with label “1111.” Thus, binary attributecognitive diagnosis models represent a subset of latent class modelswith the ability to connect class membership to item responses in amanner specified by the Q-matrix.

Methods of estimating parameter and ability values in cognitivediagnosis models include maximum likelihood (ML) optimization algorithmsand Bayesian MCMC algorithms. ML techniques include (1) Joint ML (JML),which determines the likelihood of answering an item correctly based onthe item parameters and examinee parameters; (2) Conditional ML (CML),which substitutes the total correct score for an examinee for the latentabilities of the examinee in the JML equation; and (3) Marginal ML(MML), which performs integration over the examinee attribute for modelstoo complex for computation using the CML algorithm.

MCMC algorithms estimate the set of item (b) and latent examinee (θ)parameters by using a stationary Markov chain, (A⁰, A¹, A², . . . ),with A^(t)=(b^(t), θ^(t)). The individual steps of the chain aredetermined according to the transition kernel, which is the probabilityof a transition from state t to state t+1, P[(b^(t+1), θ^(t+1))|(b^(t),d^(t))]. The goal of the MCMC algorithm is to use a transition kernelthat will allow sampling from the posterior distribution of interest.The process of sampling from the posterior distribution can be evaluatedby sampling from the distribution of each of the different types ofparameters separately. Furthermore, each of the individual elements ofthe vector can be sampled separately. Accordingly, the posteriordistribution to be sampled for the item parameters is P(b_(j)|X, θ)(across all j) and the posterior distribution to be sampled for theexaminee parameters is P(θ_(i)|X, b) (across all i).

One problem with MCMC algorithms is that the choice of a proposaldistribution is critical to the number of iterations required forconvergence of the Markov Chain. A critical measure of effectiveness ofthe choice of proposal distribution is the proportion of proposals thatare accepted within the chain. If the proportion is low, then manyunreasonable values are proposed, and the chain moves very slowlytowards convergence. Likewise, if the proportion is very high, thevalues proposed are too close to the values of the current state, andthe chain will converge very slowly.

While MCMC algorithms suffer from the same pitfalls of JML optimizationalgorithms, such as no guarantee of consistent parameter estimates, apotential strength of the MCMC approaches is the reporting of examinee(binary) attribute estimates as posterior probabilities. Thus, MCMCalgorithms can provide a more practical way of investigating cognitivediagnosis models.

Different methods of sampling values from the complete conditionaldistributions of the parameters of the model include the Gibbs samplingalgorithm and the Metropolis-Hastings within Gibbs (MHG) algorithm. Eachof the cognitive diagnosis models fit with MCMC used the MHG algorithmto evaluate the set of examinee variables because the Gibbs samplingalgorithm requires the computation of a normalizing constant. Adisadvantage of the MHG algorithm is that the set of examinee parametersare considered within a single block (i.e., only one parameter isvariable while other variables are fixed). While the use of blockingspeeds up the convergence of the MCMC chain, efficiency may be reduced.For example, attributes with large influences on the likelihood mayovershadow values of individual attributes that are not as large.

What is needed is a method for performing cognitive diagnosis thatevaluates examinees on individual skills using an overall skill ability.

A further need exists for a method for evaluating examinees onindividual skills using covariates.

A further need exists for a method that considers each attributeseparately when assessing examinees.

A still further need exists for a method of classifies examinees usingpolytomous attribute skill levels.

The present invention is directed to solving one or more of the problemsdescribed above.

SUMMARY

Before the present methods, systems and materials are described, it isto be understood that this invention is not limited to the particularmethodologies, systems and materials described, as these may vary. It isalso to be understood that the terminology used in the description isfor the purpose of describing the particular versions or embodimentsonly, and is not intended to limit the scope of the invention which willbe limited only by the appended claims.

It must also be noted that as used herein and in the appended claims,the singular forms “a,” “an,” and “the” include plural references unlessthe context clearly dictates otherwise. Thus, for example, reference toan “assessment item” is a reference to one or more assessment items andequivalents thereof known to those skilled in the art, and so forth.Unless defined otherwise, all technical and scientific terms used hereinhave the same meanings as commonly understood by one of ordinary skillin the art. Although any methods, materials, and devices similar orequivalent to those described herein can be used in the practice ortesting of embodiments of the invention, the preferred methods,materials, and devices are now described. All publications mentionedherein are incorporated by reference. Nothing herein is to be construedas an admission that the invention is not entitled to antedate suchdisclosure by virtue of prior distribution invention.

In an embodiment, a method for determining one or more examineeattribute mastery levels from an assessment includes receiving anoverall skill level for an examinee, and, for each of one or moreattributes, receiving a weight for the overall skill level, computing anexaminee attribute value based on at least the overall skill level, theweight corresponding to the attribute and one or more responses made bythe examinee to one or more questions pertaining to the attribute on theassessment, and assigning an examinee attribute mastery level for theexaminee with respect to the attribute based on whether the examineeattribute value surpasses one or more thresholds. The overall skilllevel is a rating of the examinee's performance on at least a portion ofan assessment. Computing an examinee attribute value may includecomputing a product of the overall skill level and the weight, andsumming the product and a value based on the one or more responses.

In an embodiment, the method further includes receiving a covariatevector for an examinee, and receiving, for each attribute, a weightingvector for the covariate vector. The covariate vector includes a valuefor each of one or more covariates for the examinee. The examineeattribute value is further based on the covariate vector and theweighting vector. Computing an examinee attribute value may includecomputing a product of the overall skill level and the weight, computinga vector product of the covariate vector and the weighting vector forthe attribute, and summing the product, the vector product and a valuebased on the one or more responses.

In an embodiment, a method for determining one or more examineeattribute mastery levels from an assessment includes receiving acovariate vector for an examinee, and, for each of one or moreattributes, receiving a weighting vector for the covariate vector,computing an examinee attribute value based on at least the covariatevector, the weighting vector and one or more responses made by theexaminee to one or more questions pertaining to the attribute on anassessment, and assigning an examinee attribute mastery level for theexaminee with respect to the attribute based on whether the examineeattribute value surpasses one or more thresholds. The covariate vectorincludes a value for each of one or more covariates for the examinee.Computing an examinee attribute value may include computing a vectorproduct of the covariate vector and the weighting vector, and summingthe vector product and a value based on the one or more responses.

In an embodiment, a method for determining examinee attribute masterylevels includes, for each item on an assessment, determining anestimated value for each of one or more item parameters, for eachproficiency space parameter, determining an estimated value for theproficiency space parameter, for each examinee parameter, determining anestimated value for the examinee parameter for each examinee,determining each item parameter, proficiency space parameter andexaminee parameter a predetermined number of times, and determining oneor more examinee attribute mastery levels based on the item parameters,proficiency space parameters, and examinee parameters.

In an embodiment, determining an estimated value for each of one or moreitem parameters includes performing a Metropolis-Hastings within Gibbsstep for updating an estimate of a probability of correctly applying allattributes described in a Q-matrix for an item if each attribute for theitem is mastered, performing a Metropolis-Hastings within Gibbs step forupdating an estimate of a penalty imposed on the probability ofcorrectly applying all attributes in the Q-matrix if an examinee has notmastered a particular attribute, and performing a Metropolis-Hastingswithin Gibbs step for updating an estimate of a measure of whether theattributes in the Q-matrix adequately describe attributes necessary foranswering the item.

In an embodiment, determining an estimated value for each of one or moreitem parameters further includes for each parameter specified by theQ-matrix, performing a Metropolis-Hastings within Gibbs step forupdating an estimate of a penalty exponent, which links an examineeattribute mastery level and a Q-matrix entry, for a penalty imposed onthe probability of correctly applying all attributes in the Q-matrix ifan examinee has not mastered a particular attribute, and determining anestimated value for each proficiency space parameter includes for eachattribute, performing a Metropolis-Hastings within Gibbs step forupdating an estimate of an overall skill level weighting factor, foreach attribute and each of one or more mastery ranges, performing aMetropolis-Hastings within Gibbs step for updating an estimate for eachof one or more mastery thresholds, for each examinee, performing aMetropolis-Hastings within Gibbs step for updating an estimate of anoverall skill level for an examinee, and for each covariate, attributeand continuous examinee parameter, performing a Metropolis-Hastingswithin Gibbs step for updating an estimate for each element of acovariate weighting vector.

In an embodiment, determining an estimated value for each proficiencyspace parameter includes for each attribute, performing aMetropolis-Hastings within Gibbs step for updating an estimate of anoverall skill level weighting factor, for each attribute, performing aMetropolis-Hastings within Gibbs step for updating an estimate of amastery threshold, for each examinee, performing a Metropolis-Hastingswithin Gibbs step for updating an estimate of an overall skill level forthe examinee, and for each covariate, attribute and continuous examineeparameter, performing a Metropolis-Hastings within Gibbs step forupdating an estimate of each element of a covariate weighting vector.

In an embodiment, determining an estimated value for each examineeparameter includes, for each examinee performing a Gibbs step forupdating an estimate of an examinee attribute mastery value, andperforming a Metropolis-Hastings within Gibbs step for updating anestimate of a latent ability value.

In an embodiment, a method of updating an estimate of a value from amoving window distribution includes receiving a prior estimate of avalue, receiving a base interval for a moving window distribution havingan absolute lower limit and an absolute upper limit, receiving a widthfor the moving window distribution, determining a minimum selectablevalue equal to the maximum of i) the prior estimate of the value minushalf of the width of the moving window distribution and ii) the absolutelower limit, determining a maximum selectable value equal to the minimumof i) the prior estimate of the value plus half of the width of themoving window distribution and ii) the absolute upper limit, selectingthe estimate for the value from a uniform distribution between theminimum selectable value and the maximum selectable value.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and form a part ofthe specification, illustrate various embodiments and, together with thedescription, serve to explain the principles of the various embodiments.

FIG. 1 depicts an exemplary probabilistic structure for an item responseaccording to an embodiment.

FIG. 2 depicts an exemplary hierarchical Bayesian model for the examineeparameters according to an embodiment.

FIG. 3 depicts an exemplary hierarchical Bayesian model according to anembodiment.

FIG. 4 depicts an exemplary flowchart for estimating one or more itemparameters according to an embodiment.

FIG. 5 depicts an exemplary flowchart for estimating one or moreproficiency space parameters and one or more examinee parametersaccording to an embodiment.

FIG. 6 depicts an exemplary flowchart for estimating one or more itemparameters according to an embodiment.

FIG. 7 depicts an exemplary flowchart for estimating one or moreproficiency space parameters and one or more examinee parametersaccording to an embodiment.

DETAILED DESCRIPTION

The present invention relates to a method and system for evaluatingassessment examinees on a plurality of attributes based on responses toassessment items.

Considering each attribute separately, the full conditional distributionfor a Gibbs sampling step for each attribute may be computed. Theposterior distribution for each attribute may be a Bernoullidistribution, such that P(α_(ik)=1|X_(i)=x_(i), Ψ)˜B(1, ν_(ik)), where Ψis the set of all variables excluding the attribute α_(ik). For example,under the RUM, the vector Ψ may include all of the item parameters, thecontinuous examinee parameter θ_(i), and all other attributes:Ψ=(π₁*,π₂*, . . . ,π_(J)*,r₁₁*,r₂₁*, . . . ,r_(JK)*, c₁,c₂, . . .,c_(J),θ_(i),α_(i,l≠k))′.

The probability of attribute mastery, ν_(ik), may be the fullconditional probability that examinee i is a master of attribute k.Using Bayes' Theorem, this probability is equivalent to${{P\left( {{\alpha_{ik} = {1❘x_{i}}},\Psi} \right)} = \frac{{P\left( {{{x_{i}❘\alpha_{ik}} = 1},\Psi} \right)}{P\left( {\alpha_{ik} = 1} \right)}}{\sum\limits_{l = 0}^{1}{{P\left( {{{x_{i}❘\alpha_{ik}} = l},\Psi} \right)}{P\left( {\alpha_{ik} = l} \right)}}}},$where P(α_(ik)=1) is the prior distribution of α_(ik).

The Gibbs examinee attribute parameter step detailed below may eliminatethe need for the set of underlying continuous variables required by theFusion Model. While the RUM is used as the cognitive diagnosis model inthe description contained herein, the algorithm presented is notexclusive to the RUM. Any cognitive diagnosis model may use a similarGibbs sampling procedure to estimate the attribute patterns ofexaminees.

The conditional probability that examinee i is a master of attribute k,ν_(ik), may be central to the Gibbs sampling estimation procedure.ν_(ik) may be conditional on the data and all other parameters in themodel. As defined, the prior distribution for attribute mastery may bemodeled in a plurality of different ways, such as by using a higherorder trait model or examinee covariate information such as gender,class year or location of school district.

Examinee covariates may be used in the estimation of traditional itemresponse theory (“IRT”) models. When estimated simultaneously withexaminee variables, covariates may indirectly improve the accuracy ofitem parameter estimates.

One type of examinee level covariate is the higher order latent trait.Higher order latent traits may be fit with single and multiple strategyDINA and reparameterized NIDA models. The higher order latent trait maymodel the joint distribution of the attributes. A logistic link functionmay be used to model the attributes, such as:${{P\left( {\alpha_{ik}❘G_{i}} \right)} = \frac{{\mathbb{e}}^{({\lambda_{0k} + {\lambda_{1k}G}})}}{1 + {\mathbb{e}}^{({\lambda_{0k} + {\lambda_{1k}G}})}}},$where α_(ik) is the value of the k^(th) attribute for the i^(th)examinee, G_(i) is the continuous higher order trait for examinee i(i.e., the overall ability of examinee i), and (λ_(0k), λ_(1k)) are theparameters linking the k^(th) attribute to the higher order trait. Theparameters (λ_(0k), λ_(1k)) may be similar to the item parameters of a2PL model.

Information regarding the characteristics of the proficiency space ofthe examinee parameters may be needed. The levels of mastery of a set ofdichotomous attributes prevalent in a population may be particularlyinformative. Moreover, the correlation structure of the attributesdefined by a Q-matrix may provide information regarding the nature ofthe association of the examinee attribute parameters. Additionally, inlarge scale administrations of achievement tests, subpopulationperformance characteristics (such as examining if men and women performdifferently) may be of great concern. Modeling the proficiency space togain information regarding its critical features may be performed usinga hierarchical Bayesian approach of estimating the hyperparameters ofthe distribution of examinee parameters.

The generalized linear mixed model may be used to determine populationmastery, proficiency space correlation and covariate inclusion. Ageneralized linear mixed model parameterization for modeling theproficiency space expressing the probability that examinee i is a masterof attribute k may be linearly modeled as {overscore(α)}_(ik)=β_(k)Y_(i)+λ_(k)G_(i)+e_(ik), where I(α_(ik)=1)=I({overscore(α)}_(ik)>κ_(k)) and κ_(k) is the attribute-specific cut point (masterythreshold) that determines the population proportion of masters. Theelements of vector β_(k) are the weights of the corresponding covariatescontained in the vector Y_(i). The parameter λ_(k) is the loading ofattribute k onto the higher order trait G_(i). λ_(k) may range between(−1, 1), although positive values are expected in the context of mentaltraits. e_(ik) includes error terms having a form N(0, 1−λ_(k) ²) andindependent of G_(i) and Y_(i) for all i. Accordingly, P({overscore(α)}_(ik)|G_(i)=g_(i), Y_(i)=y_(i))˜N(β_(k)y_(i)+λ_(k)g_(i),1−λ_(k) ²),where g_(i) is the realization of the higher order trait G_(i). Thehierarchical Bayesian prior distribution is thus: $\begin{matrix}{{P\left( {{\alpha_{ik} = {{1❘G_{i}} = g_{i}}},{Y_{i} = y_{i}}} \right)} = {P\left( {{{{\overset{\_}{\alpha}}_{ik} > \kappa_{k}}❘g_{i}},y_{i}} \right)}} \\{{= {\Phi\left( \frac{{\beta_{k}y_{i}} + {\lambda_{k}g_{i}} - \kappa_{k}}{\sqrt{1 - \lambda_{k}^{2}}} \right)}},}\end{matrix}$where Φ(•) is the standard normal cumulative distribution function(“CDF”).

Similarly, the RUM features a continuous examinee parameter, θ_(i),which may be modeled as θ_(i)=β_(θ)Y_(i)+λ_(θG) _(i)+e_(iθ) andP(θ|G_(i)=g_(i), Y_(i)=y_(i))˜N(β_(θ)y_(i)+λ_(θ)g_(i),1−λ_(θ) ²).

The algorithm for the estimation of the RUM using the generalized linearmixed proficiency space model may include modeling item parameters,proficiency space parameters and examinee parameters. In each of apredetermined number of iterations, each of the item parameters may beestimated. Then, each of the parameters in the proficiency space modelmay be estimated before the examinee parameters (α_(ik) and θ_(i)) areestimated for each examinee.

In an embodiment, a model MCMC algorithm uses both Gibbs sampling (forthe examinee attribute parameters) and the Metropolis-Hastings withinGibbs algorithm (MHG). Other embodiments of the MCMC algorithm may useonly the Gibbs sampling or MHG for all parameters or use Gibbs samplingor MHG for different parameters than in the described MCMC algorithm.For each MHG step, candidate values may be drawn from a moving windowfamily of proposal distributions.

A moving window proposal distribution may generate efficient proposalswith easily computed transition probability ratios. The proposal valuefor step t, with specified maximum width w on the interval [a, b], isdefined as τ˜U(L_(t), U_(t)) where$L_{t} = {{{\max\left( {{\tau^{t - 1} - \frac{w}{2}},a} \right)}\quad{and}\quad W_{t}} = {{\min\left( {{\tau^{t - 1} + \frac{w}{2}},b} \right)}.}}$Thus, the proposal distribution is a family of uniform distributionscentered at τ^(t−1) if the whole width is included in the parameterspace, and is asymmetric if not. If the distribution is asymmetric, theratio of the transition probabilities is the ratio of the heights of theuniform rectangles.

Under MHG, for a given item parameter τ, the probability of acceptanceof a candidate parameter τ* is min(1, r_(MH)), where $\begin{matrix}{r_{MH} = {\prod\limits_{i = 1}^{I}{\prod\limits_{j = 1}^{J}{\frac{\left\lfloor {\left( {P_{RUM}\left( \tau^{*} \right)} \right)^{X_{ij}}\left( {1 - {P_{RUM}\left( \tau^{*} \right)}} \right)^{({1 - X_{ij}})}} \right\rfloor{P\left( \tau^{*} \right)}{Q\left( {\tau_{t - 1}❘\tau^{*}} \right)}}{\left\lfloor {\left( {P_{RUM}\left( \tau_{t - 1} \right)} \right)^{X_{ij}}\left( {1 - {P_{RUM}\left( \tau_{t - 1} \right)}} \right)^{({1 - X_{ij}})}} \right\rfloor{P\left( \tau_{t - 1} \right)}{Q\left( {\tau^{*}❘\tau_{t - 1}} \right)}}.}}}} & \left( {{Eqn}.\quad 1} \right)\end{matrix}$τ^(t−1) is the value of the item parameter from the previous step in thechain. The function P_(RUM)(τ) is the likelihood value of an itemresponse function, such as${\pi_{j}^{*}{\prod\limits_{k = 1}^{K}{r_{jk}^{*{({1 - \alpha_{ik}})}{xq}_{jk}}{P_{c_{j}}\left( \theta_{i} \right)}}}},$using the data and the parameters specified in the chain. The valuesQ(τ_(t−1)|τ*) and Q(τ|*τ_(t−1)) are the proposal probabilities given bythe ratio of heights of the uniform proposal distributions. The valuesP(τ*) and P(τt−1) are the prior distribution probabilities of theparameters.

The model algorithm's item parameter meta-step (performed over all itemsj) may include: (1) a MHG step for π_(j)*, (2) a MHG step for r_(jk)*,for each parameter separately, with parameters specified by theQ-matrix, and (3) a MHG step for c_(j).

The MHG step for π_(j)* may include drawing π_(l)* from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor π_(j)* is U(0, 1). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for r_(jk)* may include drawing r_(jk)* from a uniformmoving window proposal distribution, U(L_(t), U_(t)). The priordistribution for r_(jk)* is U(0, 1). The candidate parameters may beaccepted with probability min(1, r_(MH)) where r_(MH) is defined in Eqn.1.

The MHG step for c_(j) may include drawing c_(j) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor c_(j) may be U(0, 3). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The model algorithm's examinee and proficiency space parameter meta-stepmay include: (1) a MHG step for λ_(k), for each attribute separately;(2) a MHG step for κ_(k), for each attribute separately; (3) a MHG stepfor g_(i), for each examinee i; (4) a MHG step for β_(k1), separatelyfor each covariate l, attribute k, and θ; (5) a Gibbs step for eachattribute parameter α_(ik), for each examinee i; and (6) a MHG step forθ_(i), for each examinee i.

The MHG step for λ_(k) may include drawing λ_(k) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor λ_(k) may be U(−1, 1). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for κ_(k) may include drawing κ_(k) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor κ_(k) may be U(−4, 4). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for g_(i) may include drawing g_(i) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor g_(i) may be N(0, 1). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for β_(k) may include drawing β_(k) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor β_(k) may be U(−4, 4). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The prior distribution for α_(ik) may be$\Phi\quad{\left( \frac{{\beta_{k}y_{i}} + {\lambda_{k}g_{i}} - \kappa_{k}}{\sqrt{1 - \lambda_{k}^{2}}} \right).}$α_(ik) may be drawn from a Bernoulli distribution with probability ofsuccess equal to$\frac{P\quad\left( {{\left. x_{i} \middle| \alpha_{ik} \right. = 1},\Psi} \right)\quad P\quad\left( {\alpha_{ik} = 1} \right)}{\sum\limits_{l = 0}^{1}\quad{P\quad\left( {{\left. x_{i} \middle| \alpha_{ik} \right. = l},\Psi} \right)\quad P\quad\left( {\alpha_{ik} = l} \right)}}.$

The MHG step for θ_(i) may include drawing θ_(i) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor θ_(i) may be N(β_(θ)y_(i)+λ_(θ)g_(i), 1−λ_(θ) ²). The candidateparameters may be accepted with probability min(1, r_(MH)) where r_(MH)is defined in Eqn. 1.

In alternate embodiments of the above-described MCMC algorithm, one ormore of β, λ and κ may be set to zero. If β=0, the fourth step of theexaminee/proficiency space meta-step need be skipped. If λ=0, the firststep of the examinee/proficiency space meta-step may be skipped. If κ=0,the second step of the examinee/proficiency space meta-step may beskipped. If β=0 and λ=0, the first and fourth steps of theexaminee/proficiency space meta-step may be skipped, and the priordistributions for α_(ik) and θ_(i) equal Φ(κ_(k)) and N(0, 1),respectively.

In an alternate embodiment, a model incorporating polytomous attributesis described. Current models for cognitive diagnosis do not allow forpolytomous attributes (attribute values other than zero or one). Whilemost of these models work well with dichotomous attributes, situationsoccur, in practice, where attributes are not dichotomous. For example,if examinees are classified into three levels for meeting educationalstandards, such as “below standards,” “meeting standards” and “exceedingstandards,” present cognitive diagnosis models cannot handle such aclassification. Standards-based classification of this sort is typicallyperformed at the level of specific content areas, such as mathematics,although any and all uses are encompassed within the scope of thepresent invention. In the context of cognitive diagnosis, thisclassification may be considered to create trichotomous attributes,where a score of zero on an attribute may indicate that an examinee isbelow the standard, a score of one may indicate that the examinee meetsthe standard, and a score of two may indicate that the examinee exceedsthe standard.

The above-described example is common in standards-based assessments. Animportant characteristic of the classification may be that the threeclasses for each attribute form an ordered set in which examinees whoare in the highest class also possess the attribute levels guaranteed byeach of the lower classes. In other words, the skill level implied by aclass exceeds that of all lesser classes. More or fewer classes may beused in a polytomous attribute classification.

A method incorporating polytomous attributes may be generated, forexample, by defining a set of general functions that relate the itemresponse function to the level of the attribute. For example, if amethod of describing the item-attribute level relationship as a functionof the level of the examinee attribute α_(ik), and the Q-matrix entryfor the j^(th) attribute, q_(jk), defined generally as f_(jk)(α_(ik),q_(jk)). For the RUM, a natural transition exists from f_(jk)(α_(ik),q_(ik)) defined for dichotomous attributes to f_(jk)(α_(ik), q_(ik))defined for polytomous attributes. Accordingly, the polytomous attributeversion of the RUM is defined as${P\quad\left( {{X_{ij} = \left. 1 \middle| \alpha_{i} \right.},\theta_{i}} \right)} = {\pi_{j}^{*}{\prod\limits_{k = 1}^{K}\quad{r_{jk}^{*{f_{jk}{({\alpha_{ik},q_{jk}})}}}{{P_{c_{j}}\left( \theta_{i} \right)}.}}}}$

The general nature of f_(jk)(α_(ik), q_(ik)) may allow for any type offunction of attribute and Q-matrix entry to affect the model likelihood.In an embodiment, the model assumes that each attribute k has discretelevels 0, 1, . . . , p. Restrictions may be placed on the model toincorporate the ordered nature of the classification. For example, suchrestrictions may include the following:f _(ik)(α_(ik)=0, q _(jk)=1)=1;   (1)f _(jk)(α_(ik) =p, q _(jk)=1)=0; and   (2)f _(jk)(α_(ik)=1, q _(jk)=1)>f _(jk)(α_(ik)=2, q _(jk)=1)> . . . >f_(jk)(α_(ik) =p−1, q _(jk)=1).   (3)

Conditions (1) and (2) may define the upper and lower limits of theattribute function. Condition (3), a monotonic decreasing ordering ofthe attribute function, may define the structure of the relationshipbetween the attribute levels and the item response function. Examineeshaving attribute level zero may have the complete application of the r*reduction to the πr parameter. In other words, such examinees may beguessing without any knowledge of the attribute. Examinees havingattribute level p may not have a reduction to the πr parameter,indicating complete knowledge of the attribute. Examinees having anattribute level z between zero and p have a decreased reduction of π*.In other words, the item response probability may increase as theattribute level increases.

Another benefit to the parameterization of the polytomous attribute RUMmay be the ability to incorporate an ordered polytomous attributestructure using a small number of parameters. f_(jk)(α_(ik), q_(jk))requires p−1 additional parameters for each Q-matrix entry. Dependingupon the number of entries in the Q-matrix, the number of f_(jk)(α_(ik),q_(jk)) parameters may become very large. In an embodiment, theconstraint: f_(1k)(α_(ik)=p, q_(ik)=1)=f_(2k)(α_(ik)=p, q_(jk)=1)= . . .=f_(jk)(α_(ik)=p, q_(ik)=1)∀p≠{0, l} my be used to provide a method ofincorporating polytomous attributes using a single parameter perattribute level.

The process of modeling polytomous attributes may include a model forthe proficiency space. The incorporation of ordered polytomousattributes may require an additional set of proficiency spaceparameters. Specifically, the ordered polytomous attribute analog of thedichotomous model presented above may include a set of p−1 cut pointparameters for each attribute, denoted by the second subscript inκ_(kp). As with the dichotomous model, {overscore(α)}_(ik)=β_(k)Y_(i)+λ_(k)G_(i)+e_(ik), where I(α_(ik)=p)=I({overscore(α)}_(ik)>κ_(kl)) and κ_(kl) is the attribute level-specific cut pointthat determines the population proportion of examinees at level z. Theelements of vector β_(k) are the weights of the corresponding covariatescontained in the vector Y_(i). The parameter λ_(k) is the loading ofattribute k onto the higher order trait G_(i). λ_(k) may range between(−1, 1), although positive values are expected in the context of mentaltraits. e_(ik) includes error terms having a form N(0, 1−λ_(k) ²) whichare independent of G_(i) and Y_(i) for all i. Accordingly, P({overscore(α)}_(ik)|G_(i)=g_(i), Y_(i)=y_(i))˜N(β_(k)y_(i)+λ_(k)g_(i), 1−λ_(k) ²),where g_(i) is the realization of the higher order trait G_(i). Thehierarchical Bayesian prior distribution is thus: $\begin{matrix}{{{P\quad\left( {{\alpha_{ik} = {\left. l \middle| G_{i} \right. = g_{i}}},{Y_{i} = y_{i}}} \right)} = \begin{matrix}{{P\left( {\left. {{\overset{\_}{\alpha}}_{ik} > \kappa_{kl}} \middle| g_{i} \right.,y_{i}} \right)} -} \\{P\quad\left( {\left. {{\overset{\_}{\alpha}}_{ik} > \kappa_{k{({l + 1})}}} \middle| g_{i} \right.,y_{i}} \right)}\end{matrix}}\quad} \\{{= {{\Phi\quad\left( \frac{{\beta_{k}y_{i}} + {\lambda_{k}g_{i}} - \kappa_{k\quad{({l + 1})}}}{\sqrt{1 - \lambda_{k}^{2}}} \right)} - {\Phi\quad\left( \frac{{\beta_{k}y_{i}} + {\lambda_{k}g_{i}} - \kappa_{kl}}{\sqrt{1 - \lambda_{k}^{2}}} \right)}}},}\end{matrix}$where Φ(•) is the standard normal CDF.

The likelihood values of the polytomous attributes are defined by thelocation of continuous variable {overscore (α)}_(ik) in relation to aset of threshold parameters, κ_(k0), κ_(k1), . . . , κ_(kp), where thereare p attribute levels for attribute k. By definition, κ_(k0)=−∞ andκ_(k(p+1))=∞. The incorporation of multiple levels for each attributemay render the current parameterization as a model for the polychoriccorrelations between the attribute parameters.

The RUM features a continuous examinee parameter, θ_(i), which may bemodeled as θ_(i)=β_(θ)Y_(i)+λ_(θ)G_(i)+e_(iθ) and P(θ|G_(i)=g_(i),Y_(i)=y_(i))˜N(β_(θ)y_(i)+λ_(θ)g_(i),1−λ_(θ) ²).

The algorithm for the estimation of the RUM using the generalized linearmixed proficiency space model may include two meta-steps: modeling theitem parameters and modeling the examinee and proficiency spaceparameters. Each of the item parameters may be examined separately.Then, each of the parameters in the proficiency space model may beexamined before the examinee parameters (α_(ik) and θ_(i)) areseparately examined for each examinee.

In an embodiment, a model MCMC algorithm uses both Gibbs sampling (forthe examinee attribute parameters) and the Metropolis-Hastings withinGibbs algorithm (MHG). Other embodiments of the MCMC algorithm may useonly the Gibbs sampling or MHG for all parameters or use Gibbs samplingor MHG for different parameters than in the described MCMC algorithm.For each MHG step, candidate values may be drawn from a moving windowfamily of proposal distributions. Under MHG, for a given item parameterτ, the probability of acceptance of a candidate parameter τ* is min(1,r_(MH)), where $\begin{matrix}{{r_{MH} = {\prod\limits_{i = 1}^{I}\quad{\prod\limits_{j = 1}^{J}\quad\frac{\begin{matrix}\left\lfloor \begin{matrix}\left( {P_{RUM}\left( \tau^{*} \right)} \right)^{X_{ij}} \\\left( {1 - {P_{RUM}\left( \tau^{*} \right)}} \right)^{({1 - X_{ij}})}\end{matrix} \right\rfloor \\{P\quad\left( \tau^{*} \right)\quad Q\quad\left( \tau_{t - 1} \middle| \tau^{*} \right)}\end{matrix}}{\begin{matrix}\begin{bmatrix}\left( {P_{RUM}\left( \tau_{t - 1} \right)} \right)^{X_{ij}} \\\left( {1 - {P_{RUM}\left( \tau_{t - 1} \right)}} \right)^{({1 - X_{ij}})}\end{bmatrix} \\{P\quad\left( \tau_{t - 1} \right)\quad Q\quad\left( \tau^{*} \middle| \tau_{t - 1} \right)}\end{matrix}}}}},} & \left( {{Eqn}.\quad 1} \right)\end{matrix}$where τ_(t−1) is the value of the parameter from the previous step inthe chain. The function P_(RUM)(τ) is the likelihood value of an itemresponse function, such as${\pi_{j}^{*}{\prod\limits_{k = 1}^{K}\quad{r_{jk}^{*{({1 - \alpha_{ik}})}\quad{xq}_{jk}}{P_{c_{j}}\left( \theta_{i} \right)}}}},$using the data and the parameters specified in the chain. The valuesQ(τ_(t−1)|τ*) and Q(τ*|τ_(t−1)) are the proposal probabilities given bythe ratio of heights of the uniform proposal distributions. The valuesP(τ*) and P(τ_(t−1)) are the prior distribution probabilities of theparameters.

The model algorithm's item parameter meta-step (performed over all itemsj) may include: (1) a MHG step for π_(j)*; (2) a MHG step for r_(jk)*,for each parameter separately, with parameters specified by theQ-matrix; (3) a MHG step for c_(j); and (4) a MHG step forf_(jk)(α_(ik)=p, q_(ik)=1) for each parameter separately, withparameters specified by the Q-matrix.

The MHG step for π_(j)* may include drawing π_(j)* from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor π_(j)* is U(0, 1). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for r_(jk)* may include drawing r_(jk)* from a uniformmoving window proposal distribution, U(L_(t), U_(t)). The priordistribution for r_(jk)* is U(0, 1). The candidate parameters may beaccepted with probability min(1, r_(MH)) where r_(MH) is defined in Eqn.1.

The MHG step for c_(j) may include drawing c_(j) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor c_(j) may be U(0, 3). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for f_(jk)(α_(ik)=p, q_(ik)=1) may include drawingf_(jk)(α_(ik)=p, q_(jk)=1) from a uniform moving window proposaldistribution, U(L_(t), U_(t)). The prior distribution forf_(jk)(α_(ik)=p, q_(jk)=1) may be U(0, 1). The candidate parameters maybe accepted with probability min(1, r_(MH)) where r_(MH) is defined inEqn. 1.

The model algorithm's examinee and proficiency space parameter meta-stepmay include: (1) a MHG step for λ_(k), for each attribute separately;(2) a MHG step for κ_(kp), for each attribute separately; (3) a MHG stepfor g_(i), for each examinee i; (4) a MHG step for β_(k1), separatelyfor each covariate l, attribute k, and θ; (5) a Gibbs step for eachattribute parameter α_(ik), for each examinee i; and (6) a MHG step forθ_(i), for each examinee i.

The MHG step for λ_(k) may include drawing λ_(k) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor λ_(k) may be U(−1, 1). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for κ_(kl) may include drawing κ_(kl) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor κ_(kl) may be U(−4, 4). The candidate parameters may be acceptedwith probability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for g_(i) may include drawing g_(i) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor g_(i) may be N(0, 1). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The MHG step for β_(k) may include drawing β_(k) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor β_(k) may be U(−4, 4). The candidate parameters may be accepted withprobability min(1, r_(MH)) where r_(MH) is defined in Eqn. 1.

The prior distribution for α_(ik) may be${\Phi\quad\left( \frac{{\beta_{k}y_{i}} + {\lambda_{k}g_{i}} - \kappa_{k\quad{({l + 1})}}}{1 - \lambda_{k}^{2}} \right)} - {\Phi\quad{\left( \frac{{\beta_{k}y_{i}} + {\lambda_{k}g_{i}} - \kappa_{kl}}{1 - \lambda_{k}^{2}} \right).}}$α_(ik) may be drawn from a Bernoulli distribution with probability ofsuccess equal to${P\quad\left( {{\alpha_{ik} = \left. l \middle| x_{i} \right.},\Psi} \right)} = {\frac{P\quad\left( {{\left. x_{i} \middle| \alpha_{ik} \right. = l},\Psi} \right)\quad P\quad\left( {\alpha_{ik} = l} \right)}{\sum\limits_{z = 0}^{p}\quad{P\quad\left( {{\left. x_{i} \middle| \alpha_{ik} \right. = z},\Psi} \right)\quad P\quad\left( {\alpha_{ik} = z} \right)}}.}$

The MHG step for θ_(i) may include drawing θ_(i) from a uniform movingwindow proposal distribution, U(L_(t), U_(t)). The prior distributionfor θ_(i) may be N(β_(θ)y_(i)+λ_(θ)g_(i),1−λ_(θ) ²). The candidateparameters may be accepted with probability min(1, r_(MH)) where r_(MH)is defined in Eqn. 1.

It is to be understood that the invention is not limited in itsapplication to the details of construction and to the arrangements ofthe components set forth in this description or illustrated in thedrawings. The invention is capable of other embodiments and of beingpracticed and carried out in various ways. Hence, it is to be understoodthat the phraseology and terminology employed herein are for the purposeof description and should not be regarded as limiting.

As such, those skilled in the art will appreciate that the conceptionupon which this disclosure is based may readily be utilized as a basisfor the designing of other structures, methods, and systems for carryingout the several purposes of the present invention. It is important,therefore, that the claims be regarded as including such equivalentconstructions insofar as they do not depart from the spirit and scope ofthe present invention.

1. A method for determining one or more examinee attribute masterylevels from an assessment, the method comprising: receiving an overallskill level for an examinee, wherein the overall skill level comprises arating of the examinee's performance on at least a portion of anassessment; and for each of one or more attributes: receiving a weightfor the overall skill level, computing an examinee attribute value basedon at least the overall skill level, the weight corresponding to theattribute and one or more responses made by the examinee to one or morequestions pertaining to the attribute on the assessment, and assigningan examinee attribute mastery level for the examinee with respect to theattribute based on whether the examinee attribute value surpasses one ormore thresholds.
 2. The method of claim 1 wherein computing an examineeattribute value comprises: computing a product of the overall skilllevel and the weight; and summing the product and a value based on theone or more responses.
 3. The method of claim 1, further comprising:receiving a covariate vector for an examinee, wherein the covariatevector includes a value for each of one or more covariates for theexaminee; and receiving, for each attribute, a weighting vector for thecovariate vector, wherein the examinee attribute value is further basedon the covariate vector and the weighting vector.
 4. The method of claim3 wherein computing an examinee attribute value comprises: computing aproduct of the overall skill level and the weight; computing a vectorproduct of the covariate vector and the weighting vector for theattribute; and summing the product, the vector product and a value basedon the one or more responses.
 5. A method for determining one or moreexaminee attribute mastery levels from an assessment, the methodcomprising: receiving a covariate vector for an examinee, wherein thecovariate vector includes a value for each of one or more covariates forthe examinee; and for each of one or more attributes: receiving aweighting vector for the covariate vector, computing an examineeattribute value based on at least the covariate vector, the weightingvector and one or more responses made by the examinee to one or morequestions pertaining to the attribute on an assessment, and assigning anexaminee attribute mastery level for the examinee with respect to theattribute based on whether the examinee attribute value surpasses one ormore thresholds.
 6. The method of claim 5 wherein computing an examineeattribute value comprises: computing a vector product of the covariatevector and the weighting vector; and summing the vector product and avalue based on the one or more responses.
 7. A method for determiningexaminee attribute mastery levels, the method comprising: (a) for eachitem on an assessment, determining an estimated value for each of one ormore item parameters; (b) for each proficiency space parameter,determining an estimated value for the proficiency space parameter; (c)for each examinee parameter, determining an estimated value for theexaminee parameter for each examinee; (d) repeating (a) through (c) apredetermined number of iterations; and (e) determining one or moreexaminee attribute mastery levels based on the item parameters,proficiency space parameters, and examinee parameters.
 8. The method ofclaim 7 wherein step (a) comprises: performing a Metropolis-Hastingswithin Gibbs step for updating an estimate of a probability of correctlyapplying all attributes described in a Q-matrix for an item if eachattribute for the item is mastered; performing a Metropolis-Hastingswithin Gibbs step for updating an estimate of a penalty imposed on theprobability of correctly applying all attributes in the Q-matrix if anexaminee has not mastered a particular attribute; and performing aMetropolis-Hastings within Gibbs step for updating an estimate of ameasure of whether the attributes in the Q-matrix adequately describeattributes necessary for answering the item.
 9. The method of claim 8wherein performing a Metropolis-Hastings within Gibbs step for updatingan estimate of a probability comprises: selecting a value for theprobability from a moving window distribution.
 10. The method of claim 8wherein performing a Metropolis-Hastings within Gibbs step for updatingan estimate of a probability comprises: accepting a value for theestimated probability with a likelihood that is the minimum of one andthe product, for all items and examinees, of i) either a) a probabilityof an examinee correctly answering an item with the estimatedprobability if the examinee answered the item correctly or b) one minusa probability of the examinee incorrectly answering the item with theestimated probability if the examinee answered the item incorrectly, ii)a prior probability of the estimated probability and iii) a firstproposal probability divided by the product, for all items andexaminees, of iv) either a) a probability of the examinee correctlyanswering an item with a previous computed probability if the examineeanswered the item correctly or b) one minus a probability of theexaminee incorrectly answering the item with the previous computedprobability if the examinee answered the item incorrectly, v) a priorprobability of the previous computed probability and vi) a secondproposal probability.
 11. The method of claim 8 wherein the priordistribution for an estimated probability comprises a uniformdistribution between 0 and
 1. 12. The method of claim 8 whereinperforming a Metropolis-Hastings within Gibbs step for updating anestimate of a penalty comprises: selecting a value for the estimatedpenalty from a moving window distribution.
 13. The method of claim 8wherein performing a Metropolis-Hastings within Gibbs step for updatingan estimate of a penalty comprises: accepting a value for the estimatedpenalty with a likelihood that is the minimum of one and the product,for all items and examinees, of i) either a) a probability of anexaminee correctly answering an item with the estimated penalty if theexaminee answered the item correctly or b) one minus a probability ofthe examinee incorrectly answering the item with the estimated penaltyif the examinee answered the item incorrectly, ii) a prior probabilityof the estimated penalty and iii) a first proposal probability dividedby the product, for all items and examinees, of iv) either a) aprobability of the examinee correctly answering an item with a previouscomputed penalty if the examinee answered the item correctly or b) oneminus a probability of the examinee incorrectly answering the item withthe previous computed penalty if the examinee answered the itemincorrectly, v) a prior probability of the previous computed penalty andvi) a second proposal probability.
 14. The method of claim 8 wherein theprior distribution for an estimated penalty comprises a uniformdistribution between 0 and
 1. 15. The method of claim 8 whereinperforming a Metropolis-Hastings within Gibbs step for updating anestimate of a measure comprises: selecting a value for the estimatedmeasure from a moving window distribution.
 16. The method of claim 8wherein performing a Metropolis-Hastings within Gibbs step for updatingan estimate of a measure comprises: accepting a value for the estimatedmeasure with a likelihood that is the minimum of one and the product,for all items and examinees, of i) either a) a probability of anexaminee correctly answering an item with the estimated measure if theexaminee answered the item correctly or b) one minus a probability ofthe examinee incorrectly answering the item with the estimated measureif the examinee answered the item incorrectly, ii) a prior probabilityof the estimated measure and iii) a first proposal probability dividedby the product, for all items and examinees, of iv) either a) aprobability of the examinee correctly answering an item with a previouscomputed measure if the examinee answered the item correctly or b) oneminus a probability of the examinee incorrectly answering the item withthe previous computed measure if the examinee answered the itemincorrectly, v) a prior probability of the previous computed measure andvi) a second proposal probability.
 17. The method of claim 8 wherein theprior distribution for an estimated measure comprises a uniformdistribution between 0 and
 3. 18. The method of claim 8 wherein step (a)further comprises: for each parameter specified by the Q-matrix,performing a Metropolis-Hastings within Gibbs step for updating anestimate of a penalty exponent, which links an examinee attributemastery level and a Q-matrix entry, for a penalty imposed on theprobability of correctly applying all attributes in the Q-matrix if anexaminee has not mastered a particular attribute, and wherein step (c)comprises: for each attribute, performing a Metropolis-Hastings withinGibbs step for updating an estimate of an overall skill level weightingfactor; for each attribute and each of one or more mastery ranges,performing a Metropolis-Hastings within Gibbs step for updating anestimate for each of one or more mastery thresholds; for each examinee,performing a Metropolis-Hastings within Gibbs step for updating anestimate of an overall skill level for an examinee; and for eachcovariate, attribute and continuous examinee parameter, performing aMetropolis-Hastings within Gibbs step for updating an estimate for eachelement of a covariate weighting vector.
 19. The method of claim 18wherein performing a Metropolis-Hastings within Gibbs step for updatingan estimate of a penalty exponent comprises: selecting an estimatedpenalty exponent from a moving window distribution.
 20. The method ofclaim 18 wherein performing a Metropolis-Hastings within Gibbs step forupdating an estimate of a penalty exponent further comprises: acceptinga value for the estimated penalty exponent with a likelihood that is theminimum of one and the product, for all items and examinees, of i)either a) a probability of an examinee correctly answering an item withthe estimated penalty exponent if the examinee answered the itemcorrectly or b) one minus a probability of the examinee incorrectlyanswering the item with the estimated penalty exponent if the examineeanswered the item incorrectly, ii) a prior probability of the estimatedpenalty exponent and iii) a first proposal probability divided by theproduct, for all items and examinees, of iv) either a) a probability ofthe examinee correctly answering an item with a previous computedpenalty exponent if the examinee answered the item correctly or b) oneminus a probability of the examinee incorrectly answering the item withthe previous computed penalty exponent if the examinee answered the itemincorrectly, v) a prior probability of the previous computed penaltyexponent and vi) a second proposal probability.
 21. The method of claim18 wherein the prior distribution for an estimated penalty exponentcomprises a uniform distribution from 0 to
 1. 22. The method of claim 18wherein performing a Metropolis-Hastings within Gibbs step for updatingan estimate of an overall skill level weighting factor comprises:selecting an estimated overall skill level weighting factor from amoving window distribution.
 23. The method of claim 18 whereinperforming a Metropolis-Hastings within Gibbs step for updating anestimate of an overall skill level weighting factor comprises: acceptinga value for the estimated skill level weighting factor with a likelihoodthat is the minimum of one and the product, for all items and examinees,of i) either a) a probability of an examinee correctly answering an itemwith the estimated skill level weighting factor if the examinee answeredthe item correctly or b) one minus a probability of the examineeincorrectly answering the item with the estimated skill level weightingfactor if the examinee answered the item incorrectly, ii) a priorprobability of the estimated skill level weighting factor and iii) afirst proposal probability divided by the product, for all items andexaminees, of iv) either a) a probability of the examinee correctlyanswering an item with a previous computed skill level weighting factorif the examinee answered the item correctly or b) one minus aprobability of the examinee incorrectly answering the item with theprevious computed skill level weighting factor if the examinee answeredthe item incorrectly, v) a prior probability of the previous computedskill level weighting factor and vi) a second proposal probability. 24.The method of claim 18 wherein the prior distribution for an estimatedoverall skill level weighting factor comprises a uniform distributionbetween −1 and
 1. 25. The method of claim 18 wherein performing aMetropolis-Hastings within Gibbs step for updating an estimate of one ormore mastery thresholds comprises: for each mastery threshold, selectingan estimated mastery threshold from a moving window distribution. 26.The method of claim 18 wherein performing a Metropolis-Hastings withinGibbs step for updating an estimate of one or more mastery thresholdscomprises: accepting a value for the estimated mastery threshold with alikelihood that is the minimum of one and the product, for all items andexaminees, of i) either a) a probability of an examinee correctlyanswering an item with the estimated mastery threshold if the examineeanswered the item correctly or b) one minus a probability of theexaminee incorrectly answering the item with the estimated masterythreshold if the examinee answered the item incorrectly, ii) a priorprobability of the estimated mastery threshold and iii) a first proposalprobability divided by the product, for all items and examinees, of iv)either a) a probability of the examinee correctly answering an item witha previous computed mastery threshold if the examinee answered the itemcorrectly or b) one minus a probability of the examinee incorrectlyanswering the item with the previous computed mastery threshold if theexaminee answered the item incorrectly, v) a prior probability of theprevious computed mastery threshold and vi) a second proposalprobability.
 27. The method of claim 18 wherein the prior distributionfor each estimated mastery threshold comprises a uniform distributionbetween −4 and
 4. 28. The method of claim 18 wherein performing aMetropolis-Hastings within Gibbs step for updating an estimate of anoverall skill level for the examinee comprises: selecting an estimatedoverall skill level from a moving window distribution.
 29. The method ofclaim 18 wherein performing a Metropolis-Hastings within Gibbs step forupdating an estimate of an overall skill level for the examineecomprises: accepting a value for the estimated overall skill level witha likelihood that is the minimum of one and the product, for all itemsand examinees, of i) either a) a probability of an examinee correctlyanswering an item with the estimated overall skill level if the examineeanswered the item correctly or b) one minus a probability of theexaminee incorrectly answering the item with the estimated overall skilllevel if the examinee answered the item incorrectly, ii) a priorprobability of the estimated overall skill level and iii) a firstproposal probability divided by the product, for all items andexaminees, of iv) either a) a probability of the examinee correctlyanswering an item with a previous computed overall skill level if theexaminee answered the item correctly or b) one minus a probability ofthe examinee incorrectly answering the item with the previous computedoverall skill level if the examinee answered the item incorrectly, v) aprior probability of the previous computed overall skill level and vi) asecond proposal probability.
 30. The method of claim 18 wherein theprior distribution for an estimated overall skill level comprises anormal distribution having a mean equal to 0 and a variance equal to 1.31. The method of claim 18 wherein performing a Metropolis-Hastingswithin Gibbs step for updating an estimate for each element of acovariate weighting vector comprises: for each element of a covariateweighting vector, selecting an estimate for the element from a movingwindow distribution.
 32. The method of claim 18 wherein performing aMetropolis-Hastings within Gibbs step for updating an estimate for eachelement of a covariate weighting vector comprises: accepting an estimatefor an element of a covariate weighting vector with a likelihood that isthe minimum of one and the product, for all items and examinees, of i)either a) a probability of an examinee correctly answering an item withthe estimated element if the examinee answered the item correctly or b)one minus a probability of the examinee incorrectly answering the itemwith the estimated element if the examinee answered the itemincorrectly, ii) a prior probability of the estimated element and iii) afirst proposal probability divided by the product, for all items andexaminees, of iv) either a) a probability of the examinee correctlyanswering an item with a previous computed element if the examineeanswered the item correctly or b) one minus a probability of theexaminee incorrectly answering the item with the previous computedelement if the examinee answered the item incorrectly, v) a priorprobability of the previous computed element and vi) a second proposalprobability.
 33. The method of claim 18 wherein a prior distribution forthe estimated element of the covariate weighting vector comprises auniform distribution between −4 and
 4. 34. The method of claim 7 whereinstep (c) comprises: for each attribute, performing a Metropolis-Hastingswithin Gibbs step for updating an estimate of an overall skill levelweighting factor; for each attribute, performing a Metropolis-Hastingswithin Gibbs step for updating an estimate of a mastery threshold; foreach examinee, performing a Metropolis-Hastings within Gibbs step forupdating an estimate of an overall skill level for the examinee; and foreach covariate, attribute and continuous examinee parameter, performinga Metropolis-Hastings within Gibbs step for updating an estimate of eachelement of a covariate weighting vector.
 35. The method of claim 34wherein performing a Metropolis-Hastings within Gibbs step for updatingan estimate of an overall skill level weighting factor comprises:selecting an estimated overall skill level weighting factor from amoving window distribution.
 36. The method of claim 34 whereinperforming a Metropolis-Hastings within Gibbs step for updating anestimate of an overall skill level weighting factor comprises: acceptinga value for the estimated overall skill level weighting factor with alikelihood that is the minimum of one and the product, for all items andexaminees, of i) either a) a probability of an examinee correctlyanswering an item with the estimated overall skill level weightingfactor if the examinee answered the item correctly or b) one minus aprobability of the examinee incorrectly answering the item with theestimated overall skill level weighting factor if the examinee answeredthe item incorrectly, ii) a prior probability of the estimated overallskill level weighting factor and iii) a first proposal probabilitydivided by the product, for all items and examinees, of iv) either a) aprobability of the examinee correctly answering an item with a previouscomputed overall skill level weighting factor if the examinee answeredthe item correctly or b) one minus a probability of the examineeincorrectly answering the item with the previous computed overall skilllevel weighting factor if the examinee answered the item incorrectly, v)a prior probability of the previous computed overall skill levelweighting factor and vi) a second proposal probability.
 37. The methodof claim 34 wherein the prior distribution for an estimated overallskill level weighting factor comprises a uniform distribution between −1and
 1. 38. The method of claim 34 wherein performing aMetropolis-Hastings within Gibbs step for updating an estimate for eachof one or more mastery thresholds comprises: selecting, for each masterythreshold, an estimated mastery threshold from a moving windowdistribution.
 39. The method of claim 34 wherein performing aMetropolis-Hastings within Gibbs step for updating an estimate for eachof one or more mastery thresholds comprises: for each mastery threshold,accepting a value for the estimated mastery threshold with a likelihoodthat is the minimum of one and the product, for all items and examinees,of i) either a) a probability of an examinee correctly answering an itemwith the estimated mastery threshold if the examinee answered the itemcorrectly or b) one minus a probability of the examinee incorrectlyanswering the item with the estimated mastery threshold if the examineeanswered the item incorrectly, ii) a prior probability of the estimatedmastery threshold and iii) a first proposal probability divided by theproduct, for all items and examinees, of iv) either a) a probability ofthe examinee correctly answering an item with a previous computedmastery threshold if the examinee answered the item correctly or b) oneminus a probability of the examinee incorrectly answering the item withthe previous computed mastery threshold if the examinee answered theitem incorrectly, v) a prior probability of the previous computedmastery threshold and vi) a second proposal probability.
 40. The methodof claim 34 wherein the prior distribution for each estimated masterythreshold comprises a uniform distribution between −4 and
 4. 41. Themethod of claim 34 wherein performing a Metropolis-Hastings within Gibbsstep for updating an estimate of an overall skill level for the examineecomprises: selecting an estimated overall skill level from a movingwindow distribution.
 42. The method of claim 34 wherein performing aMetropolis-Hastings within Gibbs step for updating an estimate of anoverall skill level for the examinee comprises: accepting a value forthe estimated overall skill level with a likelihood that is the minimumof one and the product, for all items and examinees, of i) either a) aprobability of an examinee correctly answering an item with theestimated overall skill level if the examinee answered the itemcorrectly or b) one minus a probability of the examinee incorrectlyanswering the item with the estimated overall skill level if theexaminee answered the item incorrectly, ii) a prior probability of theestimated overall skill level and iii) a first proposal probabilitydivided by the product, for all items and examinees, of iv) either a) aprobability of the examinee correctly answering an item with a previouscomputed overall skill level if the examinee answered the item correctlyor b) one minus a probability of the examinee incorrectly answering theitem with the previous computed overall skill level if the examineeanswered the item incorrectly, v) a prior probability of the previouscomputed overall skill level and vi) a second proposal probability. 43.The method of claim 34 wherein the prior distribution for an estimatedoverall skill level comprises a normal distribution having a mean equalto 0 and a variance equal to
 1. 44. The method of claim 34 whereinperforming a Metropolis-Hastings within Gibbs step for updating anestimate of each element of a covariate weighting vector comprises: foreach element of a covariate weighting vector, selecting an estimate forthe element from a moving window distribution.
 45. The method of claim34 wherein performing a Metropolis-Hastings within Gibbs step forupdating an estimate of each element of a covariate weighting vectorcomprises: accepting an estimate for an element of a covariate weightingvector with a likelihood that is the minimum of one and the product, forall items and examinees, of i) either a) a probability of an examineecorrectly answering an item with the estimated element if the examineeanswered the item correctly or b) one minus a probability of theexaminee incorrectly answering the item with the estimated element ifthe examinee answered the item incorrectly, ii) a prior probability ofthe estimated element and iii) a first proposal probability divided bythe product, for all items and examinees, of iv) either a) a probabilityof the examinee correctly answering an item with a previous computedelement if the examinee answered the item correctly or b) one minus aprobability of the examinee incorrectly answering the item with theprevious computed element if the examinee answered the item incorrectly,v) a prior probability of the previous computed element and vi) a secondproposal probability.
 46. The method of claim 34 wherein a priordistribution for the estimated element of the covariate weighting vectorcomprises a uniform distribution between −4 and
 4. 47. The method ofclaim 7 wherein step (d) comprises, for each examinee: performing aGibbs step for updating an estimate of an examinee attribute masterylevel; and performing a Metropolis-Hastings within Gibbs step forupdating an estimate of a latent ability value.
 48. The method of claim47 wherein performing the Gibbs step comprises: selecting an examineeattribute value from a normal distribution having a mean equal to thesum of a vector product of the covariate weighting vector and acovariate vector and a product of the overall skill level weightingfactor and the overall skill level, and a variance equal to one minusthe overall skill level weighting factor squared; and determining anexaminee attribute master level based on the examinee attribute valueand one or more mastery thresholds.
 49. The method of claim 47 whereinthe prior distribution for an estimated examinee attribute mastery levelcomprises a standard normal cumulative distribution function of aproduct of a covariate weighting factor vector and a covariate vectorplus the product of an overall skill value weighting factor and anoverall skill value minus a mastery threshold all divided by the squareroot of one minus the overall skill value weighting factor squared. 50.The method of claim 49 wherein the examinee attribute mastery level isdrawn from a Bernoulli distribution with probability of success equal tothe product of a probability that the examinee answers a questioncorrectly given that the examinee is a master of the particularattribute, a set of item parameters and attribute parameters and theprior distribution that the examinee is a master of the particularattribute divided by the sum of i) the product of a probability that theexaminee answers the question correctly given that the examinee is not amaster of the particular attribute, a set of item parameters andattribute parameters and the prior distribution that the examinee is nota master of the particular attribute and ii) the product of aprobability that the examinee answers a question correctly given thatthe examinee is a master of the particular attribute, a set of itemparameters and attribute parameters and the prior distribution that theexaminee is a master of the particular attribute.
 51. The method ofclaim 47 wherein the prior distribution for an estimated examineeattribute mastery level comprises the difference between i) a standardnormal cumulative distribution function of a product of a covariateweighting factor vector and a covariate vector plus the product of anoverall skill value weighting factor and an overall skill value minus afirst mastery threshold all divided by the square root of one minus theoverall skill value weighting factor squared and ii) a standard normalcumulative distribution function of a product of the covariate weightingfactor vector and the covariate vector plus the product of the overallskill value weighting factor and the overall skill value minus a secondmastery threshold all divided by the square root of one minus theoverall skill value weighting factor squared.
 52. The method of claim 51wherein the examinee attribute mastery level is drawn from a Bernoullidistribution with probability of success equal to the product of aprobability that an examinee answers a question correctly given that theparticular attribute is mastered for a set of item parameters andattribute parameters and the prior distribution for the estimatedexaminee attribute mastery level divided by the sum of the products, foreach examinee attribute mastery level, of the probability that theexaminee answers the question correctly if the examinee has the examineeattribute mastery level, a set of item parameters and attributeparameters and the prior distribution for the examinee attribute masterylevel.
 53. The method of claim 47 wherein performing theMetropolis-Hastings within Gibbs step comprises: selecting an estimatedlatent ability value from a moving window distribution.
 54. The methodof claim 47 wherein performing the Metropolis-Hastings within Gibbs stepcomprises: accepting a value for the estimated latent ability value witha likelihood that is the minimum of one and the product, for all itemsand examinees, of i) either a) a probability of an examinee correctlyanswering an item with the estimated latent ability value if theexaminee answered the item correctly or b) one minus a probability ofthe examinee incorrectly answering the item with the estimated latentability value if the examinee answered the item incorrectly, ii) a priorprobability of the estimated latent ability value and iii) a firstproposal probability divided by the product, for all items andexaminees, of iv) either a) a probability of the examinee correctlyanswering an item with a previous computed latent ability value if theexaminee answered the item correctly or b) one minus a probability ofthe examinee incorrectly answering the item with the previous computedlatent ability value if the examinee answered the item incorrectly, v) aprior probability of the previous computed latent ability value and vi)a second proposal probability.
 55. The method of claim 47 wherein theprior distribution for an estimated latent ability value comprises anormal distribution having a mean equal to the sum of the vector productof a covariate weighting factor vector and a covariate vector and theproduct of an overall skill value weighting factor and an overall skillvalue and a variance equal to one minus the overall skill valueweighting factor squared.
 56. A method of updating an estimate of avalue from a moving window distribution, the method comprising:receiving a prior estimate of a value; receiving a base interval for amoving window distribution, wherein the interval includes an absolutelower limit and an absolute upper limit; receiving a width for themoving window distribution; determining a minimum selectable value equalto the maximum of i) the prior estimate of the value minus half of thewidth of the moving window distribution and ii) the absolute lowerlimit; determining a maximum selectable value equal to the minimum of i)the prior estimate of the value plus half of the width of the movingwindow distribution and ii) the absolute upper limit; selecting theestimate for the value from a uniform distribution between the minimumselectable value and the maximum selectable value.